Opera has launched a new feature that allows users to download and run large language models locally on their computers, with over 150 models from more than 50 families available.

The discussion surrounding the viability of AI companies, particularly those focused on large language models (LLMs), highlights the significant financial and operational challenges they face. Building these models is an expensive endeavor, with companies like OpenAI reportedly burning through billions annually to fund their research and development. As the technology evolves, the costs associated with creating new models are expected to rise, making it increasingly difficult to maintain a competitive edge. The analogy of climbing Mount Everest is used to illustrate this point: as one ascends, the challenges become greater, and the resources required to push further become more demanding. Despite these challenges, there is a strong belief in the potential of LLMs as the next big technological breakthrough. Companies are motivated by the prospect of creating artificial general intelligence and the financial rewards that could follow. However, the rapid pace of innovation means that the value of existing models diminishes quickly. For instance, if a new and improved model is released, users can easily switch to it, making it essential for companies to consistently deliver top-tier models to remain relevant. The article also contrasts the AI industry with traditional cloud service providers. While building a cloud infrastructure requires significant time and investment, creating an AI model can be achieved relatively quickly, especially if a team of skilled researchers decides to leave an established company and start anew. This creates a precarious environment for AI vendors, as their competitive advantages can be eroded swiftly. The question of what constitutes a sustainable competitive advantage for LLM vendors remains open. Brand loyalty, inertia, and the development of superior applications are potential factors, but the ongoing need for substantial investment in model improvement poses a significant risk. Smaller companies, in particular, may struggle to survive without a steady revenue stream or the ability to secure continuous funding. As the market evolves, timing becomes crucial. The current hype surrounding AI may not last indefinitely, and the companies that succeed will likely be those that can adapt to changing market conditions rather than simply being the fastest to innovate. The discussion raises important considerations about the future of AI companies and the sustainability of their business models in a rapidly changing landscape.

The paper titled "Scaling Optimal LR Across Token Horizons" explores the relationship between learning rates (LR) and token horizons in the training of large language models (LLMs). The authors, Johan Bjorck and his colleagues, highlight the importance of scaling in LLMs, which involves increasing model size, dataset size, and computational resources. However, they note that tuning hyperparameters extensively for the largest models is often economically unfeasible. As a solution, they propose inferring or transferring hyperparameters from smaller experiments to larger ones. While previous research has addressed hyperparameter transfer across different model sizes, the authors identify a gap in the literature regarding hyperparameter transfer across varying dataset sizes or token horizons. To address this, they conduct a comprehensive empirical study to understand how the optimal learning rate varies with token horizon during LLM training. Their findings reveal that the optimal learning rate significantly decreases as the token horizon increases, indicating that longer training periods require smaller learning rates. The authors further establish that the optimal learning rate adheres to a scaling law, allowing for accurate estimation of the optimal learning rate for longer training horizons based on data from shorter ones. They propose a practical rule-of-thumb for transferring learning rates across different token horizons, which can be implemented without additional overhead in current practices. Additionally, they analyze the learning rate used in the LLama-1 model, suggesting that it was set too high and estimating the potential performance loss resulting from this miscalibration. In conclusion, the authors argue that hyperparameter transfer across dataset sizes is a critical yet often overlooked aspect of LLM training, emphasizing the need for further exploration in this area to enhance model performance and efficiency.